Adaptive Co-Allocation of Distributed Resources for Parallel Applications

Parallel applications can speed up their execution by accessing resources hosted by multiple autonomous providers. Applications following the message passing model demand processors to be available at the same time; a problem known as resource coallocation. Other application models, such as workflows and bag-of-tasks (BoT), can also benefit from the coordinated allocation of resources from autonomous providers. Applications waiting for resources require constant rescheduling. However, different from single-provider settings, rescheduling across multiple providers is challenging due to the autonomy and information availability participants are willing to disclose. This thesis contributes to the area of distributed systems by proposing adaptive resource co-allocation policies for message passing and BoT applications, which aim at reducing user response time and increasing system utilisation. For message passing applications, the co-allocation policies rely on start time shifting and process remapping operations, whereas for BoT applications, the policies consider limited information access from providers and coordinated rescheduling. This thesis also shows practical deployment of the co-allocation policies in a real distributed computing environment. The four major findings of this thesis are: 1. Adaptive co-allocation for message passing applications is necessary since singlecluster applications may not fill all scheduling queue fragments generated by inaccurate run time estimates. It also allows applications to be rescheduled to a single cluster, thus eliminating inter-cluster network overhead; 2. Metaschedulers using system-generated run time estimates can reschedule applications to faster or slower resources without forcing users to overestimate execution times. Overestimations have a negative effect when scheduling parallel applications in multiple providers; 3. It is possible to keep information from providers private, such as local load and total computing power, when co-allocating resources for deadline-constrained BoT applications. Resource providers can use execution offers to advertise their interest in executing an entire BoT or only part of it without revealing private information, which is important for companies to protect their business strategies; 4. Tasks of the same BoT can be spread over time due to inaccurate run time estimates and environment heterogeneity. Coordinated rescheduling of these tasks can reduce response time for users accessing single and multiple providers. Moreover, accurate run time estimates assist metaschedulers to better distribute tasks of BoT applications on multiple providers. This is to certify that (i) the thesis comprises only my original work, (ii) due acknowledgement has been made in the text to all other material used, (iii) the thesis is less than 100,000 words in length, exclusive of table, maps, bibliographies, appendices and footnotes.